Method and system for amplifying collective intelligence using a networked hyper-swarm

ABSTRACT

Systems and methods for amplifying the intelligence of networked human populations, maximizing the accuracy of collaborative forecasts and group insights. This includes systems and methods for sending and presenting a forecasting query for a future event to a plurality of participants, each using a networked computing device, the forecasting query describing a future event to be collaboratively predicted by the population of human participants. An initial forecast response is collected from each participant and analyzed by a central server. A plurality of unique overlapping subsets of responses are determined by the server. Unique overlapping subsets of unique initial forecast responses are then displayed, at substantially the same time, on each computing device. A second forecast response is collected from each participant and a final forecast is determined.

This application is a continuation of U.S. application Ser. No.16/230,759 entitled METHOD AND SYSTEM FOR A PARALLEL DISTRIBUTEDHYPER-SWARM FOR AMPLIFYING HUMAN INTELLIGENCE, filed Dec. 21, 2018,which claims the benefit of U.S. Provisional Application No. 62/611,756entitled Method and System for a Parallel Distributed Hyper-Swarm forAmplifying Human Intelligence, filed Dec. 29, 2017, which is acontinuation-in-part of U.S. application Ser. No. 16/154,613 entitledINTERACTIVE BEHAVIORAL POLLING AND MACHINE LEARNING FOR AMPLIFICATION OFGROUP INTELLIGENCE, filed Oct. 8, 2018, claiming the benefit of U.S.Provisional Application No. 62/569,909 entitled INTERACTIVE BEHAVIORALPOLLING AND MACHINE LEARNING FOR AMPLIFICATION OF GROUP INTELLIGENCE,filed Oct. 9, 2017, which is a continuation-in-part of U.S. applicationSer. No. 16/059,698 entitled ADAPTIVE POPULATION OPTIMIZATION FORAMPLIFYING THE INTELLIGENCE OF CROWDS AND SWARMS, filed Aug. 9, 2018,claiming the benefit of U.S. Provisional Application No. 62/544,861,entitled ADAPTIVE OUTLIER ANALYSIS FOR AMPLYFYING THE INTELLIGENCE OFCROWDS AND SWARMS, filed Aug. 13, 2017 and of U.S. ProvisionalApplication No. 62/552,968 entitled SYSTEM AND METHOD FOR OPTIMIZING THEPOPULATION USED BY CROWDS AND SWARMS FOR AMPLIFIED EMERGENTINTELLIGENCE, filed Aug. 31, 2017, which is a continuation-in-part ofU.S. application Ser. No. 15/922,453 entitled PARALLELIZED SUB-FACTORAGGREGATION IN REAL-TIME SWARM-BASED COLLECTIVE INTELLIGENCE SYSTEMS,filed Mar. 15, 2018, claiming the benefit of U.S. ProvisionalApplication No. 62/473,424 entitled PARALLELIZED SUB-FACTOR AGGREGATIONIN A REAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS filed Mar. 19, 2017,which in turn is a continuation-in-part of U.S. application Ser. No.15/904,239 entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF AREMOTE VEHICLE, filed Feb. 23, 2018, now U.S. Pat. No. 10,416,666,claiming the benefit of U.S. Provisional Application No. 62/463,657entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF A ROBOTICMOBILE FIRST-PERSON STREAMING CAMERA SOURCE, filed Feb. 26, 2017 andalso claiming the benefit of U.S. Provisional Application No. 62/473,429entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF A ROBOTICMOBILE FIRST-PERSON STREAMING CAMERA SOURCE, filed Mar. 19, 2017, whichis a continuation-in-part of U.S. application Ser. No. 15/898,468entitled ADAPTIVE CONFIDENCE CALIBRATION FOR REAL-TIME SWARMINTELLIGENCE SYSTEMS, filed Feb. 17, 2018, now U.S. Pat. No. 10,712,929,claiming the benefit of U.S. Provisional Application No. 62/460,861entitled ARTIFICIAL SWARM INTELLIGENCE WITH ADAPTIVE CONFIDENCECALIBRATION, filed Feb. 19, 2017 and also claiming the benefit of U.S.Provisional Application No. 62/473,442 entitled ARTIFICIAL SWARMINTELLIGENCE WITH ADAPTIVE CONFIDENCE CALIBRATION, filed Mar. 19, 2017,which is a continuation-in-part of U.S. application Ser. No. 15/815,579entitled SYSTEMS AND METHODS FOR HYBRID SWARM INTELLIGENCE, filed Nov.16, 2017, now U.S. Pat. No. 10,439,836, claiming the benefit of U.S.Provisional Application No. 62/423,402 entitled SYSTEM AND METHOD FORHYBRID SWARM INTELLIGENCE filed Nov. 17, 2016, which is acontinuation-in-part of U.S. application Ser. No. 15/640,145 entitledMETHODS AND SYSTEMS FOR MODIFYING USER INFLUENCE DURING A COLLABORATIVESESSION OF REAL-TIME COLLABORATIVE INTELLIGENCE, filed Jun. 30, 2017,now U.S. Pat. No. 10,353,551, claiming the benefit of U.S. ProvisionalApplication No. 62/358,026 entitled METHODS AND SYSTEMS FOR AMPLIFYINGTHE INTELLIGENCE OF A HUMAN-BASED ARTIFICIAL SWARM INTELLIGENCE filedJul. 3, 2016, which is a continuation-in-part of U.S. application Ser.No. 15/241,340 entitled METHODS FOR ANALYZING DECISIONS MADE BYREAL-TIME INTELLIGENCE SYSTEMS, filed Aug. 19, 2016, now U.S. Pat. No.10,222,961, claiming the benefit of U.S. Provisional Application No.62/207,234 entitled METHODS FOR ANALYZING THE DECISIONS MADE BYREAL-TIME COLLECTIVE INTELLIGENCE SYSTEMS filed Aug. 19, 2015, which isa continuation-in-part of U.S. application Ser. No. 15/199,990 entitledMETHODS AND SYSTEMS FOR ENABLING A CREDIT ECONOMY IN A REAL-TIMECOLLABORATIVE INTELLIGENCE, filed Jul. 1, 2016, claiming the benefit ofU.S. Provisional Application No. 62/187,470 entitled METHODS AND SYSTEMSFOR ENABLING A CREDIT ECONOMY IN A REAL-TIME SYNCHRONOUS COLLABORATIVESYSTEM filed Jul. 1, 2015, which is a continuation-in-part of U.S.application Ser. No. 15/086,034 entitled SYSTEM AND METHOD FORMODERATING REAL-TIME CLOSED-LOOP COLLABORATIVE DECISIONS ON MOBILEDEVICES, filed Mar. 30, 2016, now U.S. Pat. No. 10,310,802, claiming thebenefit of U.S. Provisional Application No. 62/140,032 entitled SYSTEMAND METHOD FOR MODERATING A REAL-TIME CLOSED-LOOP COLLABORATIVE APPROVALFROM A GROUP OF MOBILE USERS filed Mar. 30, 2015, which is acontinuation-in-part of U.S. patent application Ser. No. 15/052,876,filed Feb. 25, 2016, entitled DYNAMIC SYSTEMS FOR OPTIMIZATION OFREAL-TIME COLLABORATIVE INTELLIGENCE, now U.S. Pat. No. 10,110,664,claiming the benefit of U.S. Provisional Application No. 62/120,618entitled APPLICATION OF DYNAMIC RESTORING FORCES TO OPTIMIZE GROUPINTELLIGENCE IN REAL-TIME SOCIAL SWARMS, filed Feb. 25, 2015, which is acontinuation-in-part of U.S. application Ser. No. 15/047,522 entitledSYSTEMS AND METHODS FOR COLLABORATIVE SYNCHRONOUS IMAGE SELECTION, filedFeb. 18, 2016, now U.S. Pat. No. 10,133,460, which in turn claims thebenefit of U.S. Provisional Application No. 62/117,808 entitled SYSTEMAND METHODS FOR COLLABORATIVE SYNCHRONOUS IMAGE SELECTION, filed Feb.18, 2015, which is a continuation-in-part of U.S. application Ser. No.15/017,424 entitled ITERATIVE SUGGESTION MODES FOR REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Feb. 5, 2016 which in turnclaims the benefit of U.S. Provisional Application No. 62/113,393entitled SYSTEMS AND METHODS FOR ENABLING SYNCHRONOUS COLLABORATIVECREATIVITY AND DECISION MAKING, filed Feb. 7, 2015, which is acontinuation-in-part of U.S. application Ser. No. 14/925,837 entitledMULTI-PHASE MULTI-GROUP SELECTION METHODS FOR REAL-TIME COLLABORATIVEINTELLIGENCE SYSTEMS, filed Oct. 28, 2015, now U.S. Pat. No. 10,551,999,which in turn claims the benefit of U.S. Provisional Application No.62/069,360 entitled SYSTEMS AND METHODS FOR ENABLING AND MODERATING AMASSIVELY-PARALLEL REAL-TIME SYNCHRONOUS COLLABORATIVESUPER-INTELLIGENCE, filed Oct. 28, 2014, which is a continuation-in-partof U.S. application Ser. No. 14/920,819 entitled SUGGESTION ANDBACKGROUND MODES FOR REAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS, filedOct. 22, 2015, now U.S. Pat. No. 10,277,645, which in turn claims thebenefit of U.S. Provisional Application No. 62/067,505 entitled SYSTEMAND METHODS FOR MODERATING REAL-TIME COLLABORATIVE DECISIONS OVER ADISTRIBUTED NETWORKS, filed Oct. 23, 2014, which is acontinuation-in-part of U.S. application Ser. No. 14/859,035 entitledSYSTEMS AND METHODS FOR ASSESSMENT AND OPTIMIZATION OF REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Sep. 18, 2015, now U.S. Pat.No. 10,122,775, which in turns claims the benefit of U.S. ProvisionalApplication No. 62/066,718 entitled SYSTEM AND METHOD FOR MODERATING ANDOPTIMIZING REAL-TIME SWARM INTELLIGENCES, filed Oct. 21, 2014, which isa continuation-in-part of U.S. patent application Ser. No. 14/738,768entitled INTUITIVE INTERFACES FOR REAL-TIME COLLABORATIVE INTELLIGENCE,filed Jun. 12, 2015, now U.S. Pat. No. 9,940,006, which in turn claimsthe benefit of U.S. Provisional Application 62/012,403 entitledINTUITIVE INTERFACE FOR REAL-TIME COLLABORATIVE CONTROL, filed Jun. 15,2014, which is a continuation-in-part of U.S. application Ser. No.14/708,038 entitled MULTI-GROUP METHODS AND SYSTEMS FOR REAL-TIMEMULTI-TIER COLLABORATIVE INTELLIGENCE, filed May 8, 2015, which in turnclaims the benefit of U.S. Provisional Application 61/991,505 entitledMETHODS AND SYSTEM FOR MULTI-TIER COLLABORATIVE INTELLIGENCE, filed May10, 2014, which is a continuation-in-part of U.S. patent applicationSer. No. 14/668,970 entitled METHODS AND SYSTEMS FOR REAL-TIMECOLLABORATIVE INTELLIGENCE, filed Mar. 25, 2015, now U.S. Pat. No.9,959,028, which in turn claims the benefit of U.S. ProvisionalApplication 61/970,885 entitled METHOD AND SYSTEM FOR ENABLING AGROUPWISE COLLABORATIVE CONSCIOUSNESS, filed Mar. 26, 2014, all of whichare incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.16/059,698 entitled ADAPTIVE POPULATION OPTIMIZATION FOR AMPLIFYING THEINTELLIGENCE OF CROWDS AND SWARMS, filed Aug. 9, 2018, claiming thebenefit of U.S. Provisional Application No. 62/544,861, entitledADAPTIVE OUTLIER ANALYSIS FOR AMPLYFYING THE INTELLIGENCE OF CROWDS ANDSWARMS, filed Aug. 13, 2017 and of U.S. Provisional Application No.62/552,968 entitled SYSTEM AND METHOD FOR OPTIMIZING THE POPULATION USEDBY CROWDS AND SWARMS FOR AMPLIFIED EMERGENT INTELLIGENCE, filed Aug. 31,2017, all of which are incorporated in their entirety herein byreference.

This application is a continuation-in-part of U.S. application Ser. No.16/154,613 entitled INTERACTIVE BEHAVIORAL POLLING AND MACHINE LEARNINGFOR AMPLIFICATION OF GROUP INTELLIGENCE, filed Oct. 8, 2018, claimingthe benefit of U.S. Provisional Application No. 62/569,909 entitledINTERACTIVE BEHAVIORAL POLLING AND MACHINE LEARNING FOR AMPLIFICATION OFGROUP INTELLIGENCE, filed Oct. 9, 2017, both of which are incorporatedin their entirety herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to systems and methods forharnessing and amplifying the combined intelligence of distributed humanpopulations. In particular, this invention relates to enabling networkedhuman groups to form real-time systems that deliberate and then convergeon optimized solutions to problems posed to the group, especiallyforecasting problems.

2. Discussion of the Related Art

The aggregation of insights from large groups of people has been shownto amplify intelligence (i.e. increase accuracy). Sometimes called theWisdom of Crowds, these methods generally use statistical averagingacross a static set of data collected from a population throughsurvey-based polling. While such methods have shown amplifications ofintelligence, a significant problem with all common Wisdom of Crowdmethods is that human participants are very poor at expressing theirinternal feelings in a consistent and repeatable manner that can beaggregated across populations. People vary wildly in the level ofconfidence (or conviction) they have in their reported answers, andtheir internal scales for reporting measures of confidence arenon-linear and very different from person to person. Simply put, peopleare highly inconsistent at “self-reporting” their internal thoughts,insights, intuitions, estimation, and opinions on surveys in way thatcaptures the magnitude of their confidence or conviction such that itcan be effectively aggregated across populations. The present inventionre-invents the polling process by enabling groups to adjust theirindividual forecasts with guidance from other individuals, therebyenabling internal confidence scales to converge on common metrics.Unlike other systems that enable feedback (like prediction markets)which enable interactions in series (one after the other), the currentinvention enables all participants to gain feedback in parallel. Thisenables a true “Hive Mind” to emerge, similar to swarms in nature,eliminating the flaw of market-based systems—serialized data whichdrives bubbles, busts, and momentum overshoots. In addition, machinelearning is employed in multiple steps in the process to (a) optimizethe initial estimates by curating the population, and (b) to optimizethe weighting of estimates based on human behaviors. By enabling thecapture of real-time dynamic behavioral data, the methods and systemsdisclosed here enable a more accurate and more consistent model of userconfidence and conviction than traditional reporting. And finally, aninnovative architecture is disclosed which enables a single largepopulation (for example 1000 participants) to be divided into a largenumber of unique sub-populations, which each converge on sub-estimatesin parallel. This greatly increases accuracy by eliminating thepossibility that all participants are updating their estimates (andexpressing behaviors) based on the same population data.

SUMMARY OF THE INVENTION

Several embodiments of the invention advantageously address the needsabove as well as other needs by providing a method forcomputer-moderated collaborative forecasting among a population of humanparticipants using a plurality of networked computing devices, themethod comprising: providing a collaboration server running acollaboration application, the collaboration server in communicationwith the plurality of the networked computing devices, each computingdevice associated with one member of the population of humanparticipants; providing a local forecasting application on eachnetworked computing device, the local forecasting application configuredfor displaying forecasting information to and collecting forecastinginput from the one member associated with that networked computingdevice; and enabling through communication between the collaborationapplication running on the collaboration server and the localforecasting applications running on each of the plurality of networkedcomputing devices, the following steps send a forecasting query to theplurality of networked computing devices, the forecasting querydescribing a future event to be collaboratively predicted by thepopulation of human participants; present, at substantially the sametime, a representation of the forecasting query to each member of thepopulation of human participants on a display of the computing deviceassociated with that member; collect an initial forecast response fromeach member of the population of human participants via a user interfaceon the computing device associated with that member; store a set ofinitial forecast responses from the population of human participants ina memory accessible by the collaboration server; identify a plurality ofunique overlapping subsets of initial forecast responses from the set ofinitial forecast responses, each overlapping subset sharing at least onedata point with another overlapping subset; display, at substantiallythe same time, a different unique overlapping subset of initial forecastresponses to each member of the population of human participants on thecomputing device associated with that member, thereby enabling thatmember to consider the initial forecasts responses provided by adifferent unique subset of human participants from the full populationof human participant; collect an updated forecast response from eachmember of the population of human participants via a user interface onthe computing device associated with that member; store a set of updatedforecast responses from the population of human participants in a memoryaccessible by the collaboration server; and compute a finalcollaborative forecast based at least in part upon the set of initialforecast responses and the set of updated forecast responses, thecollaborative forecast providing an answer to the forecasting query.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of severalembodiments of the present invention will be more apparent from thefollowing more particular description thereof, presented in conjunctionwith the following drawings.

FIG. 1 is a schematic diagram of an exemplary computing deviceconfigured for use in a collaboration system.

FIG. 2 is a schematic diagram of an exemplary real-time collaborationsystem.

FIG. 3 is a flowchart of an exemplary group collaboration process usingthe collaboration system.

FIG. 4 is a schematic representation of an exemplary 3D grid hyper-swarmparticipant data structure.

FIG. 5 is an illustration of an exemplary grid-arranged populationparticipants.

FIG. 6 is a flowchart for a Hyper-Swarm Iterative ApproximationEmbodiment method.

FIG. 7 is an exemplary display of a primary forecast variable userinterface.

FIG. 8 is an exemplary stimulus display for a single participant in thepopulation.

FIG. 9 is an updated version of the exemplary stimulus display of FIG.8.

FIG. 10 is a version of the exemplary stimulus display of FIG. 8including weighting factors.

FIG. 11 is an exemplary stimulus display including a histogram.

FIG. 12 is an exemplary volumetric visualization of hyper-swarmcharacteristics.

FIG. 13 is a flowchart for a real-time swarming embodiment method.

FIG. 14 is an illustration of an exemplary grid-arranged population withparticipants grouped in regions.

FIG. 15 is an illustration of an exemplary supervisory subswarmgrid-arranged embodiment.

FIG. 16 is an exemplary slider set user interface display for amulti-option question.

FIG. 17 is an illustration of an exemplary visualization display for themulti-option question.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings. Skilled artisans willappreciate that elements in the figures are illustrated for simplicityand clarity and have not necessarily been drawn to scale. For example,the dimensions of some of the elements in the figures may be exaggeratedrelative to other elements to help to improve understanding of variousembodiments of the present invention. Also, common but well-understoodelements that are useful or necessary in a commercially feasibleembodiment are often not depicted in order to facilitate a lessobstructed view of these various embodiments of the present invention.

DETAILED DESCRIPTION

The following description is not to be taken in a limiting sense, but ismade merely for the purpose of describing the general principles ofexemplary embodiments. Reference throughout this specification to “oneembodiment,” “an embodiment,” or similar language means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment,”“in an embodiment,” and similar language throughout this specificationmay, but do not necessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

Real-time occurrences as referenced herein are those that aresubstantially current within the context of human perception andreaction.

As referred to in this specification, “media items” refers to video,audio, streaming and any combination thereof. In addition, the audiosubsystem is envisioned to optionally include features such as graphicequalization, volume, balance, fading, base and treble controls,surround sound emulation, and noise reduction. One skilled in therelevant art will appreciate that the above cited list of file formatsis not intended to be all inclusive.

Historical research demonstrates that the insights generated by groupscan be more accurate than the insights generated by individuals in manysituations. A classic example is estimating the number of beans in ajar. Many researchers have shown that taking the statistical average ofestimates made by many individuals will yield an answer that is moreaccurate than the typical member of the population queried. There arethree basic categories of population-based data aggregation systems:polls, markets, and swarms. Traditional Polls treat all individuals asisolated datapoints, the only interaction among participants isstatistical, when a researcher processes the data after-the-fact andcomputes an aggregated result. Polling has been shown to amplify theintelligence of a population to a moderate level. A significant drawbackof traditional polling is that there is no interaction betweenparticipants to enable them to adjust their beliefs and/or confidencelevels based upon insights gained from others. To solve this, SequentialPolls have been deployed where participants can see the results of thepoll before they respond. This has been shown to reduce accuracy becauseeach individual is influenced by the people who came before them,resulting in amplified social influence bias. This is often described asan asymmetric herding effect which makes users overcompensate fornegative ratings but amplify positive ones, accumulating in bubble thatdistorts results by over 30%. This phenomenon was first described in apaper written by Lev Muchnik, Sinan Aral and Sean J. Taylor in 2014.Similar to traditional polls, prediction markets are structured inseries, causing each transaction to influence the next transaction. Thistoo, causes significant distortions to the collective intelligence thatcan be extracted from a population, resulting in asymmetric momentumwhich leads to bubbles and busts.

To solve this problem, an alternate method of aggregation has beendeveloped and deployed called “swarming” wherein all participantsprovide input in parallel (not series), enabling people to benefit fromthe insights of others without as pronounced as herding effect. Aproblem with current swarming technologies is that (a) computationalburden limits the size of swarms, (b) the highly interactive nature ofthe interface limits deployment on small screens like phones, and (c)the real-time nature limits the opportunity of individuals to researchthe questions posed and consider response carefully. The presentinvention, referred to herein as a Hyper-Swarm solves these problems bybuilding a hybrid system that falls between a market and a swarm,turning a market in a parallel aggregation system (instead of serialaggregation) thereby eliminating herding effects caused by social bias,while maintaining the group-wise interactions that make swarms sopowerful. In addition, the present invention enables a large populationto be broken into a large set of “sub-swarms”, defined as interactivegroups that benefit from the insights of others.

Collaborative Intelligence Systems and Methods

Systems and methods for general collaborative swarm intelligence aredisclosed in the related applications incorporated by reference. Selectdisclosure from related applications is included in this section to aidin understanding of the present invention. It will be understood thatthis section is meant to give a general background only and not allparts of this general collaborative intelligence (swarm) systems andmethods section are directly applicable to the present invention.

As described in related U.S. Pat. No. 9,959,028 for METHODS AND SYSTEMSFOR REAL-TIME CLOSED-LOOP COLLABORATIVE INTELLIGENCE, by the presentinventor, and incorporated by reference, a swarm-based system andmethods have been developed that enable groups of users tocollaboratively control the motion of a graphical pointer through aunique real-time closed-loop control paradigm. In some embodiments, thecollaboratively controlled pointer is configured to empower a group ofusers to choose letters, words, numbers, phrases, and/or other choicesin response to a prompt posed simultaneously to the group. This enablesthe formation a group response that's not based on the will of anyindividual user, but rather on the collective will of the group. In thisway, the system disclosed herein enables a group of people to expressinsights as a unified intelligence, thereby making decisions, answeringquestions, rendering forecasts, and making predictions as an artificialswarm intelligence.

As described in U.S. patent application Ser. No. 14/708,038 forMULTI-GROUP METHODS AND SYSTEMS FOR REAL-TIME MULTI-TIER COLLABORATIVEINTELLIGENCE, by the current inventor and incorporated by reference,additional systems and methods have been disclosed that encourage groupsof real-time users who are answering questions as a swarm to producecoherent responses while discouraging incoherent responses. A number ofmethods were disclosed therein, including (a) Coherence Scoring, (b)Coherence Feedback, and (c) Tiered Processing. These and othertechniques greatly enhance the effectiveness of the resultingintelligence.

As described in U.S. patent application Ser. No. 14/859,035 for SYSTEMSAND METHODS FOR ASSESSMENT AND OPTIMIZATION OF REAL-TIME COLLABORATIVEINTELLIGENCE SYSTEMS, by the present inventor and incorporated byreference, a system and methods have been developed for enablingartificial swarms to modify its participant population dynamically overtime, optimizing the performance of the emergent intelligence byaltering its population makeup and/or the altering the relativeinfluence of members of within that population. In some such embodimentsthe members of a swarm can selectively eject one or more low performingmembers of that swarm (from the swarm), using the group-wisecollaborative decision-making techniques herein. As also disclosed,swarms can be configured to dynamically adjust its own makeup, not byejecting members of the swarm but by adjusting the relative weighting ofthe input received from members of the swarm. More specifically, in someembodiments, algorithms are used to increase the impact (weighting) thatsome users have upon the closed-loop motion of the pointer, whiledecreasing the impact (weighting that other users have upon theclosed-loop motion of the pointer. In this way, the swarm intelligenceis adapted over time by the underlying algorithms disclosed herein,strengthening the connections (i.e. input) with respect to the morecollaborative users, and weakening the connections with respect to theless collaborative users.

One thing that both poll-based methods and swarm-based methods have incommon when making predictions, is that a smarter population generallyresults in a smarter Collective Intelligence. As described in U.S.patent application Ser. No. 16/059,658 for ADAPTIVE POPULATIONOPTIMIZATION FOR AMPLYFYING THE INTELLIGENCE OF CROWDS AND SWARMS, bythe present inventor and incorporated by reference, methods and systemsare disclosed that enables the use of polling data to curate a refinedpopulation of people to form an emergent intelligence. While this methodis highly effective, by enabling dynamic behavioral data in the pollingprocess, far deeper and more accurate assessments of human confidenceand human conviction are attained and used to significantly improve thepopulation curation process. Specifically, this enables higher accuracywhen distinguishing members of the population of are likely to behigh-insight performers on a given prediction task as compared tomembers of the population who are likely to be low-insight performers ona given prediction task, and does so without using historical data abouttheir performance on similar tasks.

Referring first to FIG. 1, as previously disclosed in the related patentapplications, a schematic diagram of an exemplary portable computingdevice 100 configured for use in the collaboration system is shown.Shown are a central processor 102, a main memory 104, a timing circuit106, a display interface 108, a display 110, a secondary memorysubsystem 112, a hard disk drive 114, a removable storage drive 116, alogical media storage drive 118, a removable storage unit 120, acommunications interface 122, a user interface 124, a transceiver 126,an auxiliary interface 128, an auxiliary I/O port 130, communicationsinfrastructure 132, an audio subsystem 134, a microphone 136, headphones138, a tilt sensor 140, a central collaboration server 142, and acollaborative intent application 144.

Each of a plurality of portable computing devices 100, each used by oneof a plurality of users (the plurality of users also referred to as agroup), is networked in real-time to the central collaboration server(CCS) 142. In some embodiments, one of the portable computing devices100 could act as the central collaboration server 142. For the purposesof this disclosure, the central collaboration server 142 is its owncomputer system in a remote location, and not the portable computingdevice 100 of one of the users. Hence the collaboration system iscomprised of the centralized central collaboration server 142 and theplurality of portable computing devices 100, each of the portablecomputing devices 100 used by one user.

The portable computing device 100 may be embodied as a handheld unit, apocket housed unit, a body worn unit, or other portable unit that isgenerally maintained on the person of a user. The portable computingdevice 100 may be wearable, such as transmissive display glasses.

The central processor 102 is provided to interpret and execute logicalinstructions stored in the main memory 104. The main memory 104 is theprimary general purpose storage area for instructions and data to beprocessed by the central processor 102. The main memory 104 is used inthe broadest sense and may include RAM, EEPROM and ROM. The timingcircuit 106 is provided to coordinate activities within the portablecomputing device 100. The central processor 102, main memory 104 andtiming circuit 106 are directly coupled to the communicationsinfrastructure 132. The central processor 102 may be configured to run avariety of applications, including for example phone and address bookapplications, media storage and play applications, gaming applications,clock and timing applications, phone and email and text messaging andchat and other communication applications. The central processor 102 isalso configured to run at least one Collaborative Intent Application(CIA) 144. The Collaborative Intent Application 144 may be a standaloneapplication or may be a component of an application that also runs uponother networked processors.

The portable computing device 100 includes the communicationsinfrastructure 132 used to transfer data, memory addresses where dataitems are to be found and control signals among the various componentsand subsystems of the portable computing device 100.

The display interface 108 is provided upon the portable computing device100 to drive the display 110 associated with the portable computingdevice 100. The display interface 108 is electrically coupled to thecommunications infrastructure 132 and provides signals to the display110 for visually outputting both graphics and alphanumeric characters.The display interface 108 may include a dedicated graphics processor andmemory to support the displaying of graphics intensive media. Thedisplay 110 may be of any type (e.g., cathode ray tube, gas plasma) butin most circumstances will usually be a solid state device such asliquid crystal display. The display 110 may include a touch screencapability, allowing manual input as well as graphical display.

Affixed to the display 110, directly or indirectly, is the tilt sensor140 (accelerometer or other effective technology) that detects thephysical orientation of the display 110. The tilt sensor 140 is alsocoupled to the central processor 102 so that input conveyed via the tiltsensor 140 is transferred to the central processor 102. The tilt sensor140 provides input to the Collaborative Intent Application 144, asdescribed later. Other input methods may include eye tracking, voiceinput, and/or manipulandum input.

The secondary memory subsystem 112 is provided which houses retrievablestorage units such as the hard disk drive 114 and the removable storagedrive 116. Optional storage units such as the logical media storagedrive 118 and the removable storage unit 118 may also be included. Theremovable storage drive 116 may be a replaceable hard drive, opticalmedia storage drive or a solid state flash RAM device. The logical mediastorage drive 118 may be a flash RAM device, EEPROM encoded withplayable media, or optical storage media (CD, DVD). The removablestorage unit 120 may be logical, optical or of an electromechanical(hard disk) design.

The communications interface 122 subsystem is provided which allows forstandardized electrical connection of peripheral devices to thecommunications infrastructure 132 including, serial, parallel, USB, andFirewire connectivity. For example, the user interface 124 and thetransceiver 126 are electrically coupled to the communicationsinfrastructure 132 via the communications interface 122. For purposes ofthis disclosure, the term user interface 124 includes the hardware andoperating software by which the user executes procedures on the portablecomputing device 100 and the means by which the portable computingdevice 100 conveys information to the user. In some embodiments the userinterface 124 is controlled by the CIA 144 and is configured to displayinformation regarding the group collaboration, as well as receive userinput and display group output.

To accommodate non-standardized communications interfaces (i.e.,proprietary), the optional separate auxiliary interface 128 and theauxiliary I/O port 130 are provided to couple proprietary peripheraldevices to the communications infrastructure 132. The transceiver 126facilitates the remote exchange of data and synchronizing signalsbetween the portable computing device 100 and the Central CollaborationServer 142. The transceiver 126 could also be used to enablecommunication among a plurality of portable computing devices 100 usedby other participants. In some embodiments, one of the portablecomputing devices 100 acts as the Central Collaboration Server 142,although the ideal embodiment uses a dedicated server for this purpose.In one embodiment the transceiver 126 is a radio frequency type normallyassociated with computer networks for example, wireless computernetworks based on BlueTooth® or the various IEEE standards802.11.sub.x., where x denotes the various present and evolving wirelesscomputing standards. In some embodiments the portable computing devices100 establish an ad hock network between and among them, as with aBlueTooth® communication technology.

It should be noted that any prevailing wireless communication standardmay be employed to enable the plurality of portable computing devices100 to exchange data and thereby engage in a collaborative consciousnessprocess. For example, digital cellular communications formats compatiblewith for example GSM, 3G, 4G, and evolving cellular communicationsstandards. Both peer-to-peer (PPP) and client-server models areenvisioned for implementation. In a third alternative embodiment, thetransceiver 126 may include hybrids of computer communicationsstandards, cellular standards and evolving satellite radio standards.

The audio subsystem 134 is provided and electrically coupled to thecommunications infrastructure 132. The audio subsystem 134 is configuredfor the playback and recording of digital media, for example, multi ormultimedia encoded in any of the exemplary formats MP3, AVI, WAV, MPG,QT, WMA, AIFF, AU, RAM, RA, MOV, MIDI, etc.

The audio subsystem 134 in one embodiment includes the microphone 136which is used for the detection and capture of vocal utterances fromthat unit's user. In this way the user may issue a suggestion as averbal utterance. The portable computing device 100 may then capture theverbal utterance, digitize the utterance, and convey the utterance toother of said plurality of users by sending it to their respectiveportable computing devices 100 over the intervening network. In thisway, the user may convey a suggestion verbally and have the suggestionconveyed as verbal audio content to other users. It should be noted thatif the users are in close physical proximity the suggestion may beconveyed verbally without the need for conveying it through anelectronic media. The user may simply speak the suggestion to the othermembers of the group who are in close listening range. Those users maythen accept or reject the suggestion using their portable electronicdevices 100 and taking advantage of the tallying, processing, andelectronic decision determination and communication processes disclosedherein. In this way the system may act as a supportive supplement thatis seamlessly integrated into a direct face to face conversation heldamong a group of users.

For embodiments that do include the microphone 136, it may beincorporated within the casing of the portable computing device 100 ormay be remotely located elsewhere upon a body of the user and isconnected to the portable computing device 100 by a wired or wirelesslink. Sound signals from microphone 136 are generally captured as analogaudio signals and converted to digital form by an analog to digitalconverter or other similar component and/or process. A digital signal isthereby provided to the processor 102 of the portable computing device100, the digital signal representing the audio content captured bymicrophone 136. In some embodiments the microphone 136 is local to theheadphones 138 or other head-worn component of the user. In someembodiments the microphone 136 is interfaced to the portable computingdevice 100 by a Bluetooth® link. In some embodiments the microphone 136comprises a plurality of microphone elements. This can allow users totalk to each other, while engaging in a collaborative experience, makingit more fun and social. Allowing users to talk to each other could alsobe distracting and could be not allowed.

The audio subsystem 134 generally also includes headphones 138 (or othersimilar personalized audio presentation units that display audio contentto the ears of a user). The headphones 138 may be connected by wired orwireless connections. In some embodiments the headphones 138 areinterfaced to the portable computing device 100 by the Bluetooth®communication link.

The portable computing device 100 includes an operating system, thenecessary hardware and software drivers necessary to fully utilize thedevices coupled to the communications infrastructure 132, media playbackand recording applications and at least one Collaborative IntentApplication 144 operatively loaded into main memory 104, which isdesigned to display information to a user, collect input from that user,and communicate in real-time with the Central Collaboration Server 142.Optionally, the portable computing device 100 is envisioned to includeat least one remote authentication application, one or more cryptographyapplications capable of performing symmetric and asymmetriccryptographic functions, and secure messaging software. Optionally, theportable computing device 100 may be disposed in a portable form factorto be carried by a user.

Referring next to FIG. 2, an exemplary collaboration system 200 isshown. Shown are the central collaboration server 142, a plurality ofthe secondary memory subsystems 112, a plurality of the timing circuits106, a first portable computing device 202, a second portable computingdevice 204, a third portable computing device 206, and a plurality ofexchanges of data 208.

The group of users (participants), each using one of the plurality ofportable computing devices 100, each portable computing device 100running the Collaborative Intent Application 144, each device 100 incommunication with the Central Collaboration Server 142, may engage inthe collaborative experience that evokes a collective intelligence (alsoreferred to as Collective Consciousness).

As shown in FIG. 2, the CCS 142 is in communication with the pluralityof portable computing devices 202, 204, 206. Each of these devices 202,204, 206 is running the Collaborative Intent Application (CIA) 144. Inone example, each of the devices 202, 204, 206 is an iPad® running theCIA 144, each iPad® communicating with the CCS 142 which is running aCollaboration Mediation application (CMA) 210. Thus, we have the localCIA 144 on each of the plurality of devices 202, 204, 206, each device202, 204, 206 in real-time communication with the CMA running on the CCS142. While only three portable devices 202, 204, 206 are shown in FIG. 2for clarity, in ideal embodiments, dozens, hundreds, thousands, or evenmillions of devices 100 would be employed in the collaboration system200. Hence the CCS 142 must be in real-time communication with manydevices 100 at once.

The communication between the CCS 142 and each of the devices 202, 204,206 includes the exchanges of data 208. The data has a very significantreal-time function, closing the loop around each user, over theintervening electronic network.

As described above, the system 200 allows the group of users, each usingtheir own tablet or phone or other similar portable computing device100, to collaboratively answer questions in real-time with the supportof the mediating system of the CCS 142 which communicates with the localCIA 144 running on each device 100. The Collaborative Intent Application144 ties each device 100 to the overall collaborative system 200.Multiple embodiments of the CIA 144 are disclosed herein. TheCollaborative Intent Application (CIA) 144 may be architected in avariety of ways to enable the plurality of portable computing devices100 to engage in the collaborative processes described herein, with thesupportive use of the Central Collaboration Server 142.

In some embodiments the exchange of data 208 may exist between portablecomputing devices 100.

Referring next to FIG. 3, a flowchart of one embodiment of a groupcollaboration process is shown. Shown are a collaboration opportunitystep 300, a user input step 302, a send user intents to CCS step 304, adetermine group intent step 306, a send group intent to CIA step 308, adisplay intents step 310, a target selection decision point 312, and adisplay target step 314. The process also includes optional steps thatcould be included, for example, for a pointer graphical embodiment: adisplay pointer start position step 316, a display input choices step318, and an update pointer location step 320. In the collaborationopportunity step 300, the CIA 144 receives the group collaborationopportunity from the CCS 142 and displays the opportunity on the display110 of the portable computing device 100 (PCD). The group collaborationopportunity may be a question to be answered, for example, “What filmwill win the Best Picture in the Academy Awards?” or “Who will win theSuper Bowl?” The process then proceeds to the user input step 302. Theuser input step 302 includes the user using the display 110 of thecomputing device 100 to input the user intent. The user intent is aninput interpreted by the user interface 124 as a desired vectordirection conveying an intent of the user. In some embodiments(described in the related applications), the user intent is a desiredvector direction of a graphical pointer of the user interface 124, andthe user input includes swiping of the pointer via the touchscreeninterface. The user input step 302 takes place for each user of thegroup. The process then proceeds to the send user intent to CCS step304.

In the send user intent to CCS step 304, the CIA 144 for each PCD 100sends the user intent to the CCS 142. In the next step, the determinegroup intent step 306, the CCS 142 determines a collective group intentbased on the plurality of user intents. The group intent may bedetermined through various methods, as described further below. Theprocess then proceeds to the send group intent to CIA step 308.

In the embodiment including the optional steps display pointer startposition 316 and the display input choices step 318, in the displaypointer start position step 316 the graphical user interface 124 woulddisplay the starting, or neutral, position of a pointer chosen toindicate the graphical representation of the group intent. In thefollowing step, the display input choices step 318, the user interface124 would display a plurality of input choices 412 available to beselected by the group intent by using the pointer. The user intent inthis embodiment is an input interpreted by the user interface 124 asrepresenting that user's desired motion of the collaborative graphicalpointer with respect to the plurality of input choices.

In the send group intent to CIA step 308, the CIA 144 receives the groupintent from the CCS 142. Next, in the display intents step 310, for eachcomputing device 100 the received representation of the group intent isdisplayed, along with a representation of the user intent originallyinput by the user of the computing device 100. The process then proceedsto the target selection decision point 312.

The update pointer location step 320 may be inserted between the displayintents step 310 and the target selection decision point 312. In theupdate pointer location step 320, in the embodiments including thepointer the user interface 124 updates to indicate the current locationof the pointer in response to the received group intent.

In the target selection decision point 312, if the group intent receivedcorresponds to selection of the target (in some embodiments, from amongthe input choices), the process proceeds to the display target step 314,and the selected target is displayed on the display 124. If the groupintent has not selected the target, the process returns to the userinput step 302, and the process repeats until the target is determinedby the group intent or until the process is otherwise ended (forexample, by a time limit).

After the target has been chosen by the group intent, the entire processmay repeat, for example, to form a word if each consecutive target is analphabetic character.

Referring again to FIGS. 1, 2 and 3, the collaboration system in oneembodiment as previously disclosed in the related applications employsthe CCS 142 that users connect to via their portable computing device100. In some embodiments, fixed or non-portable computing devices 100can be used as well. In many embodiments, users choose or are assigned ausername when they log into the CCS 142, thus allowing software on theCCS 142 to keep track of individual users and assign each one a scorebased on their prior sessions. This also allows the CCS 142 to employuser scores when computing the average of the group intent of all theusers (in embodiments that use the average).

In general, when the session is in progress, the question is sent fromthe CCS 142 to each of the CIA 144 on the portable computing devices 100of the users. In response to the question, the users convey their ownintent either by manipulating an inner puck of the pointer, as describedin the related applications, or by using a tilt or swipe input or otheruser interface methods. In some embodiments, the user's intent isconveyed as a direction and a magnitude (a vector) that the user wantsthe pointer to move. This is a user intent vector and is conveyed to theCCS 142. In some embodiments, the magnitude of the user intent vector isconstant. The CCS 142 in some embodiments computes the numerical average(either a simple average or a weighted average) of the group intent forthe current time step. Using the numerical average, the CCS 142 updatesfor the current time step the graphical location of the pointer within atarget board displayed on the display 110. This is conveyed as anupdated coordinate location sent from the CCS 142 to each of the CIA 144of participating users on their own devices 100. This updated locationappears to each of the users on their individual devices 100. Thus theysee the moving pointer, ideally heading towards an input choice on thetarget board. The CCS 142 determines if and when the input choice issuccessfully engaged by the pointer and if so, that target is selectedas an answer, or as a part of the answer (a single letter or space orpunctuation mark, for example, that's added to an emerging answer). Thattarget is then added to the emerging answer, which is sent to all thedevices 100 and appears on each display 110.

While FIG. 3 illustrates one embodiment of the collaborative process, asshown in the related applications, many variations of the basic processare contemplated by the inventor.

Parallel Distributed Interactive Forecasting and Machine Learning

Human participants possess deep insights across a vast range of topics,having knowledge, wisdom, and intuition that can be harnesses andcombined to build an emergent collective intelligence. A significantproblem, however, people are bad at reporting the sentiments insidetheir heads and are even worse at expressing their relative levels ofconfidence and/or conviction in those sentiments. Reports from humanparticipants are inconsistent from trial to trial, using scales that arehighly non-linear, and highly inconsistent from participant toparticipant. To solve this problem, innovative methods and systems havebeen developed which expose participants to beliefs of otherparticipants and allows them to adjust their input, enabling thesoftware system to monitor how participants “behave” when making suchadjustments. This behavioral data is collected and processed which givesfar deeper insights into the true sentiments of human participants, aswell as far deeper insights into the confidence and/or conviction thatgo with the expressed sentiments. In addition, an innovative method ofbuilding sub-populations (referred to herein as sub-swarms) has beendeveloped to ensure that the population is being exposed to a diverserange of stimuli when making their adjustments, not all viewing the samestimuli. This further eliminates social biasing effects (i.e. herding)that could result from all participants being given the same global viewof population beliefs.

Unlike prediction markets and sequential polls, which are serial innature, and rely on participants showing up at a website (or otherinterface) to enter data by answering poll questions or engaging inmarket transactions, the Hyper-Swarm in preferred embodiments isstructured as a PUSH system, meaning that participants are pushed anotification or other signal that informs them that a stage of theprocess is now active, and they have a certain time window to entertheir data. This ensures that all participants are active synchronously.In some embodiments, the process is staged as a set of definedintervals. In some embodiments, real-time interaction is enabled. Thereal-time process is a true swarm, while the staged process is asimplified approximation of an actual swarm. While the real-time processis the preferred embodiment, this system will be described below firstas a staged process (iterative steps) because it's simpler, then will bedescribed as a real-time embodiment.

A good way to describe the systems and methods enabled by the hardwareand software disclosed herein is by example. We use a sports predictionexample because it's easy to explain, although the same methods andsystems can be used for financial predictions, political predictions,and other types of forecasts. The same methods and systems can also beused for capturing and amplifying the accuracy of subjective sentiments,such as views and opinions of a population. In the example used herein,the objective is to predict football games, forecasting both the winnerof the game and the number of points the winner wins by. In thisexample, a set of 15 games will be forecast by a population of 1000participant, with the objective of aggregating the data from the 1000participants to generate the most accurate group forecast possible.Traditionally, this would be done by asking a single question andcapturing static data, which has all the problems described above, or byusing a serial prediction market, which is subject to significantherding effects. In the inventive system and method, a parallel processusing a unique interface, unique prompts, and unique timer, and a uniquemachine learning process. The example we will use is a single sportsgame:

Who will win the 49ers or the Raiders, and by how many points?

The system is designed to capture forecast data from a diverse humanpopulation on a given question (the above being just one examplequestion) and process the input provided to output an aggregate forecastthat is significantly more accurate than a traditional poll, sequentialpoll, or prediction market. Two unique embodiments are described, firsta two-step iterative embodiment, then a real-time interactionembodiment. Both embodiments employ unique methods for capturing notjust a forecast, but confidence levels in the forecast. Confidence iscaptured by self-reporting, but computed by the system using bothself-reporting data and by behavioral analysis.

Referring next to FIG. 4, a schematic representation of a 3D gridhyper-swarm participant data structure 400 is shown. The data structure400 is aligned with the coordinate x-, y-, and z-axes, as shown in FIG.4.

For reasons that will become clear below, one of the preferred datastructures for a hyper-swarm is a 3D grid of participants, such thateach participant P is associated with a unique x,y,z coordinate withinthe grid. Ideally the grid is structured into a finite cube of gridpoints, and treated such that opposite faces of the cube are adjacent(i.e. if you traveled off the edge of one face you would enter theopposite face). This means that any participants who is assigned anx,y,z coordinate within the grid, has 26 neighbors within the grid. Thisis illustrated in FIG. 4, showing a selected participant P_(xyz) 402 asa large circle in the center of the grid, and the 26 smaller circles (N₁to N₂₆) located on grid intersections indicating participant neighbors404.

This means that if the full population were 1000 participants, theparticipants would be arranged such that each is assigned an XYZcoordinate on a 10×10×10 grid. And with opposite faces being treated asadjacent, every participant of the 1000 participants has 26 neighbors.In addition, we define each unique group of 27 participants (P_(xyz)plus his or her 26 neighbors) as a sub-swarm of the full population.This means there are 1000 unique Sub-swarms S₁ to S₁₀₀₀, each with 27participants, such that every sub-swarm overlaps with 26 othersub-swarms. An exemplary illustration of a grid-arranged population 500of 1000 participants with a single sub-swarm 502 highlighted is shown inFIG. 5. A participant 404 is located at each grid intersection, althoughfor clarity not all participants are indicated on FIG. 5. It's importantto note that alternate data-structures can be used to enablehyper-swarms that are consistent with the present invention. Forexample, another structure, referred to herein as a Nearest NeighborModel each of N participants are assigned a unique index in the range 1through N, and each participant is grouped with S other participantsthat have the closest indexes to them where S is a value that modulatesthe size of sub-swarms. For example, there might be 1000 participants(N=1000) and we might desire to have sub-swarms such that there are 50participants in each (S=50). Thus, in this example, each of 1000participants is assigned a unique index of 1 through N and is uniquelygrouped with the 50 other participants that have the closest indexes tothem.

Another structure, referred to herein as a Small-World ConnectivityModel is defined in two steps. First, a Nearest Neighbor Model isdefined as described above wherein each of N participants are assigned aunique index in the range 1 through N, and each participant is groupedwith S other participants that have the closest indexes to them. Then, arandomization process is used to adjust some members of each uniquesub-swarm such that they have indices that fall outside the nearestneighbor range. In one such Small-World model, the randomizationalgorithm re-assigns connections between users with 20% probability. Bythis we mean, one fifth of the members of each sub-swarm are randomlyreplaced by members who fall outside of the index range defined by the Sother participants that have the closest index. This enables a highlyefficient transfer of information throughout the 1000 participantpopulation.

Referring next to FIG. 6, a flowchart for a Hyper-Swarm IterativeApproximation Embodiment method is shown.

The Hyper-Swarm Iterative Approximation Embodiment method is structuredas an iterative forecasting method wherein all participants engage in atwo-round process, substantially synchronized (i.e. conducted inparallel) and each employing intelligent algorithms for optimization ateach round.

The first round is referred to herein as the initial forecasting roundand the second round is referred to as the secondary forcecasting round,the two rounds structured such that behavioral data is collected bycomparing the changes to made by individual participants from round toround. It should also be noted that while it's described as two roundprocess, additional rounds can be optionally implemented to capturefurther behavioral data and make further algorithmic optimization. Thenext of these additional rounds would be called a tertiary forcecstinground, etc.

This embodiment is structured as a two-dimensional forecasting method,meaning that a single forecast has two independent variables capturedfrom participants. One variable is referred to herein as a primaryforecast variable and represents a predicted outcome of an event in thefuture. The independent second variable indicates a confidence metricassociated with the first forecast variable. This is referred to as asecondary forecast variable. It should be noted that in someembodiments, additional dimensions are enabled. An example of a thirddimension is described further below.

As we are using a sports example for illustrative purposes herein,consider the primary forecast variable to be an indication of “WHO WILLWIN A FOOTBALL GAME AND BY HOW MANY POINTS?”. Thus, each of theexemplary 1000 participants would be given an opportunity provide aforecast of this primary forecast variable at each forecasting round ofthe method. In addition, consider the secondary forecast variable to bean indication of “HOW MUCH WOULD YOU BET ON YOUR FORECAST OUTCOME?” as anumerical value of confidence between $0 and $100. These two variable aslinked in that each user is asked to first make a prediction for theoutcome of the football game, and then indicate a level of wager theywould be willing to make on their forecast value. This comprises the twodimensions of forecasting.

It should be noted that this inventive system includes a scoring methodsuch that participants are awarded points based their primary andsecondary forecast variables, in consideration of the actual outcome ofthe predicted event. In one embodiment, points are awarded toparticipants based on both (a) how close to the actual outcome theirPrimary Forecast Variable was, and (b) how much they were willing towager on their Primary Forecast Variable prediction. The closer to theactual outcome the participant was, and the more they were willing towager, the more points they are awarded. In some embodiments, points areonly awarded if the Primary Forecast Variable as exactly correct—forexample, predicting the exact winner of the football game, and the exactnumber of points the winner would win by. In addition, if aparticipant's forecast is wrong (in some embodiments by being anincorrect value, or in other embodiments being off by more than athreshold amount, then instead of winning points the participants losepoints—the magnitude of the loss being based at least in part upon theSecondary Forecast Variable (i.e. the amount they we were willing towager). In some such embodiments the loss is binary, depending onwhether the primary forecast value was correct or incorrect. In otherembodiments the loss is also proportional to the degree to which theforecast value was off. In all cases, the scoring system is structure toincentivize participants to make their best possible forecast (both inprimary and secondary values, and optionally additional values) at eachround in the forecasting process.

It should be noted that scoring is based on each round of the iterativeround process. Thus, the scoring process described above would assesspoints for the initial forecasting round as well as for the secondaryforecasting round. of course, the points are not scored until later(i.e. until after the actual outcome of the event is known). In someembodiments the point value of each round is equal, for exampleparticipants having the ability to earn up to 100 points for the InitialForecasting Round, and up to 100 points for the Secondary ForecastingRound. In other embodiments, the rounds are weighted, such that oneround is worth more than another round. In a preferred embodiment, theprimary round is worth 60% of the total points, and the secondary roundis worth 40% of the total points. This weighting is used to incentivepeople to perform personal research and/or personal assessments duringthe initial round, generating the best initial forecasts possible. Forembodiments that include additional rounds, the weighting of each roundcan be equal. That said, in preferred embodiments, the weighting of eachround decreases with each iteration.

The current method can be divided into four stages. (1) SessionEngagement, (2) Round One Forecasting, (3) Round Two Forecasting, and(4) Aggregated Forecast Processing.

The session engagement stage includes a computing device notificationstep 600. The present embodiment of the Hyper-Swarm system is structuredsuch that during the computing device notification step 600 a centralserver sends out a push notification to a plurality of computingdevices, each one of the computing devices associated with one of aplurality of participants, each of the notifications indicating that anew forecast is beginning and that they will have a defined period oftime to complete their initial forecasts. In some embodiments the systemis the collaboration system 200. In some embodiments the central serveris the central collaboration server (CCS) 142. This notification can bepushed to personal computers, tablets, or phones, and is generally donethrough a traditional software application, a mobile app, or a web-appsuch that it pops up with a visual, audio, and/or tactile alert to gettheir attention.

For the sports example, a set of forecasts will be performed at the sametime, the set including a number of individual forecasts. For example,the set might be all of the football games to be played on Sunday, whichincludes forecasts for each of the individual games. As the processdescribed herein only requires a single forecast event, we will show itwith respect to just one game, but it is understood that the sameprocess is generally used for all games in the set (as independentpredictions).

In a preferred embodiment, each computing device first receives an alertinforming the associated user that a hyper-swarm session is starting andasking them to join the process. Each user can respond affirmatively ornegatively by clicking a user interface (for example, a simple buttonresponse). If the user responds affirmatively, the central server willadd the user to a data structure used for this prediction and the userwill become a participant. In the preferred embodiment, the datastructure will assign each participant a coordinate within athree-dimensional grid, as described above. The size of the grid willdepend upon the number of participants that respond affirmatively, withan upper limit cap.

In this example we will assume that the upper limit was set to 1000participants, and that the session was fully subscribed, meaning thataffirmative responses came from 1000 participants in response to thepush notifications. This means that the data structure will be definedby the central server as a 10×10×10 grid (such as that shown in FIG. 5).It also means that a first participant will be assigned the grid value(0,0,0) and a second participant will be assigned (0,0,1), and so on,until the full 10×10×10 grid is filled with participants. As describedabove, this further means that each of the 1000 participants has 26neighbors in this grid structure, using the convention that opposingfaces of the grid are adjacent (i.e. the 10th row in the 10×10 structureis adjacent to the 1st row). The reason for this structure and thedefinition of 26 neighbors will become apparent in the steps below. Itshould be noted that neighbors could be defined as a larger region, forexample, the 5×5×5 grid around a participant, which would mean 124neighbors for each person. It should noted that alternate datastructures could be used, as described previously, to associate eachparticipant with a sub-swarm. It should also be noted that the cubicstructure used in this example could further employ a Small WorldConnectivity randomization process as described previously, such thatrandom members of some sub-swarms could be replaced by members who arenot among the 26 nearest neighbors.

Regardless of how the data structure is defined, once a set ofparticipants has responding affirmatively and have been associated withthe data structure, we can proceed to the initial forecasting round.

In the next conduct initial forecasting round step 602, Each participantin the hyper-swarm session is asked to make a set of initial forecasts,for example forecasts with respect to all the football games to beplayed on a given day. They will make these forecasts one by one. Forexample purposes we will only show the details of one such forecast, theprediction of the 49ers vs. the Raiders, but it is understood that manysimilar forecasts are generally made in a single session. During theinitial forecasting round step 602, first at least one forecasting queryis sent to each of the participant networked computing devices, eachforecasting query describing a future event to be collaborativelypredicted by the population of human participants. A representation ofthe forecasting query (or queries) is presented, at substantially thesame time, to each member of the population on the computing devicedisplays.

For each of said forecast query, an interface appears for eachparticipant on their individual computing device (e.g. personalcomputer, tablet, phone). The interface can take many forms, but in apreferred embodiment, it is a set of selection lines, each with amoveable slider, one selection line/slider associated with the PrimaryForecast Variable (a primary forecast variable user interface 700 asshown in FIG. 7) and one selection line/slider associated with Secondaryforecast variable (a secondary forecast variable slider user interface702). The primary forecast selection user interface 700 includes aprimary variable selection line 720 and a primary variable slider 708that is moved along the primary variable selection line 720 by the userto select a value. As shown in FIG. 7, the current selection for theprimary variable is 0 (neutral). The secondary forecast selection userinterface 702 includes a secondary variable selection line 722 and asecondary variable slider 718. As shown in FIG. 7, the current selectionfor the secondary variable is $50. In some embodiments, additionalselection lines/sliders can be provided for addition related variablefor a single forecast. That will be shown later.

In one example embodiment, two user interfaces 700 702 appear on thescreens of all participants, at approximately the same time. Inaddition, a timer 704 appears indicating how much time the participantshave for the full set of forecasts. For a session involving forecastingall 15 football games for a given weekend, the participants may be givena 20 minute timer, for example. This enables the participants to becoordinated in time, but also gives enough time for participants toresearch their answers. The system encourages research in the firstforecasting round, as the more information participants gather (to winpoints) the smarter the overall hyper-swarm. In this way, the initialforecasting round is designed to encourage information gathering andassessment by the full population of participants.

An exemplary initial forecasting round interface is shown in FIG. 7. Inthis example the Primary forecast variable is the football team, and thesecondary forecast variable is the bet size. Each selection line 720,722 includes a first choice 710 at one end of the line 720, 722, asecond choice 712 at the opposite end of the line 720, 722, and aplurality of selection values 714 along the line 720, 722. Eachselection line 720, 722 is also associated with a prompt 716.

In addition to the user interfaces 700 and 702, a “FORECAST COMPLETE”button 706 is provided to participants, such that once they finishentering their forecast, they can register the data as final. Or theycan wait for the timer to fully expire, at which point the data is alsoregistered as final. Either way, the final values for each of thesliders above (Primary Forecast Variable user interface 700 andSecondary Forecast Variable user interface 702), i.e. an initialforecast response, is sent to the central server and is storedassociated with the given user and that user's coordinate in the gridstructure in a memory accessible by the central server (typically theCentral Collaboration Server 142). Other variables may also be stored,such as demographic data about the users age, gender, location,experience level on the subject, and self-reported skill level on thesubject in question. In some embodiments, each grid location is a blockin a blockchain structure, enabling secure data storage.

After the initial forecasting round 602 is complete, the method proceedsto the optional perform population curation analysis step 604.

The central server now has initial forecasts from all 1000 participantsin this session, for each of the forecast events (football games in thisexample) in question. Using this data, the system then performs apopulation curation analysis, determining which participants are mostlikely to be skilled forecasters and which participants are less likelyto be skilled forecasters. This process is described in co-pending U.S.application Ser. No. 16/059,698 by the present inventor and is herebyincorporated by reference. The output of this process is a weightingfactor associated with each participant, the weighting factor indicativehow likely that participants forecast values are accurate, with the mostlikely accurate forecasters being weighted higher than the least likely.In some embodiments, after the population curation analysis an optionalparticipant culling/weighting step 606 is used in addition to or insteadof the weighting process such that the weakest predicted performers areremoved from the process. In some such embodiments, if the objective isto have N participants in the hyper swarm, a set of N+M users areengaged in the initial forecasting round 602, such that M participantscan be culled based on likelihood of low performance. For example, 1100users (1000+100) could be engaged in the initial forecasting round 602,such that the lowest-performing 100 users (based on the curationprediction methods above) can be eliminated after the initialforecasting round 602, keeping only the top 1000 users. This enablesoptimization of forecasting after the initial forecasting round. Inaddition, the forecasts from the 1000 remaining users can be weightedbased upon the likelihood that they are strong forecasters. Thistwo-stage process is described in the aforementioned co-pendingapplication.

The method then proceeds to the secondary forecasting round step 608.The central server has now collected data from all participants,optionally culled participants to a smaller set of likely strongperformers, and optionally weighted remaining participants based on thepredicted likelihood that they are strong performers. The next step isfor participants to update their forecasts based on a presentedstimulus. The stimulus is an indication to each participant of how otherparticipants predicted the same events. This happens in parallel (allparticipants are informed at substantially the same time) so there isnot a sequential biasing problem. In addition, all participants areprovided with a unique but overlapping stimulus set, to ensure that adiverse range of responses is generated by the population (i.e. toensure the population is not all responding to the same stimulus). Thisis where the sub-swarm definition comes in, as during the secondaryforecasting round the central server identifies a swarm subset ofparticipants for each participant, where each subset overlapping atleast one other subset (i.e. each subset shares at least one participantwith a different subset) as each participant in the defined datastructure has a unique set of 26 neighbors. In the present embodiment,the population is arranged on nodes of a 3D grid, and the swarm subsetis defined as the participant plus all participants on neighboring nodes(including on the diagonal), as shown in FIG. 4. In this embodiment,each participant is given information to review about how their 26neighbors forecast the same event compared to their own forecast.

During the secondary forecasting round 608, at substantially the sametime, a different overlapping subset of the initial forecast responsesare displayed on each display. In the present embodiment, the initialforecast responses displayed are those of the participant plus theparticipant's 26 grid neighbors. An example stimulus display for asingle participant in the population is shown in FIG. 8. As shown inFIG. 8, each participant is provided with an inventive display ofinformation and data entry, enabling them to gain insight into thebeliefs of a sub-set of fellow participants and update their ownbeliefs. Specifically, FIG. 8 shows two-dimensional coordinate grid 804representing the two variables provided by participants, with thePrimary Forecasting Variable represented on the X axis 800 and theSecondary Forecasting Variable represented on the Y axis 802. In thisinstance, the text “Which Team will Win and by how much? (in points)”associated with the primary variable is displayed on the X axis 800, andthe text “How much would you Bet on it? (in dollars)” associated withthe secondary variable is displayed on the Y axis 802.

The data plotted on the coordinate grid 804 are unique for each andevery participant. First, what is plotted is that participant's ownforecast data (Primary and Secondary forecasting variables) as shown bythe participant data point 808. In this case we see that the location ofthe participant data point 808 on the coordinate grid 804 shows that theindividual user in question predicted the 49ers would win by 9 points(on the X axis 800) and chose to bet $50 on that forecast (on the y axis802).

Also plotted on this coordinate system are similar X-Y data points forall 26 of that participant's neighbors, as defined by that participant'spositioning within the 10×10×10 data grid described previously. In thisexample figure, the neighbor forecast data points 810 (both in score anddollars bet) are shown for these 26 other participants.

Also provided is a countdown timer 812. This time provides a time limiton how long this particular participant has to consider the displayedbeliefs of other participants, and optionally update his or her ownforecast.

To update the forecast, the participant can use the displayed sliders708, 718, moved along the corresponding selection line 720, 722, toadjust either or both variables. Alternatively, the user can drag thegraphical datapoint 808 representing his or her on forecast.

After the participant uses the user interfaces 700, 702 to update theforecast, the stimulus display is updated. An example of an updatedstimulus display is shown in FIG. 9. An updated participant data point900 showing the participant's updated forecast will indicate theparticipant's updated position, enabling the user to still see their oldprediction still shown by the participant data point 808. This ishelpful so they know their prior belief. This is also important becauseit reminds the participant that both their initial forecast and theirfinal forecast are used in the scoring system. This is illustrated inFIG. 9, which shows that at 33 seconds left on the clock 812, thisparticular participant updated their forecast to 49ers win by 4, andincreased their bet to $65 as shown by the coordinate location of theupdated participant data point 900.

During the secondary forecasting round 608, updated forecast responsesof the participants are sent to the central server and stored on amemory accessible by the central server (Central Collaboration Server142).

Secondary forecasting round step 608 in the process is extremelypowerful as it provides three critical elements. First, it enables allparticipants, in parallel, to re-evaluate their beliefs with the benefitof limited social influence (in this case, just 26 same data points outof a population of 1000). Second, it enables data to be collectedrepresenting this informative behavioral response from all participants,as the direction and magnitude of the change is indicative of confidence(as will be described below). Third, by virtue of every participantbeing exposed to a unique sub-swarm of beliefs, a diverse range ofbehaviors are induced rather than the typically flawed serial marketprocess where every participant is being influenced by the exact sametime history of individual data transactions. Thus the shift to parallelassessments (instead of serial) and the shift to distributed and uniquesub-swarms (instead of identical time histories) enable us to evoke asignificantly more insightful and accurate aggregation of knowledge,wisdom, opinions and intuitions than other methods.

Optionally an additional facet of data display is implemented, as shownin an exemplary display of FIG. 10. Specifically, for each user in thepopulation curation analysis step 604, the weighting factor is generatedbased on their full set of initial responses and the machine-learnedalgorithm for predicting the likelihood that each participant is askilled forecaster. This weighting factor is then applied to theneighbor forecast data points 810. This information is useful and can bedisplayed by varying the size and/or color of the displayed data pointsshown. Other information can also be used in determining the graphicalweighting of the data points (for example historical accuracy ofparticipants).

An example of displayed weighting factors by participant is shown inFIG. 10. As seen, by modifying the size of the neighbor forecast datapoints 810 by indicating the weighting factors associated withindividual participants as larger data point size for higher weightingfactor (indicating higher predicted likelihood that the given user isaccurate), the participant who is viewing this particular screen (of hisor her 26 neighbors) is given a more informative set of data toconsider. And with this additional information, the example user makes adifferent behavioral change to his or her initial prediction. Thisdisplay option of varying the dot size (or color, for example, instead)for the neighbor forecast data points 810 is useful for situations wherea wide range of weighting factors are generated.

In some embodiments, a participant has the option of choosing analternate data display rather than the data points (i.e. dots) displayedon the graph above. In one such embodiment, the users can display ahistogram that represents the beliefs of their sub-swarm. This is mostuseful for embodiments of much larger populations, for example 1 millionusers, represented in a 100×100×100 grid, and provided data about 7×7×7neighbors (343 neighbors). For such an example, the dots would beoverwhelming, but a histogram is informative as shown in FIG. 11. Asshown in FIG. 11, the histogram 1100 shows the dollars bet (y-axis 802)per point spread (x-axis 800). This can be a true histogram, or inpreferred embodiments us a weighted histogram using the populationcuration weighting values described above.

The method then proceeds to the secondary round analysis step 610. Afterthe initial forecasting round 602 and the secondary forecasting round608, the central server now has both initial forecasts and updatedforecasts from all 1000 participants in this session, for each of theforecast events (i.e. each of the football games). Using this data, thecentral server can now perform a second level of optimization whereinthe behaviors of users, performed in response to the unique sub-swarmdata displayed to them, is used to compute Round Two weighting factorsindicative of an inferred confidence level of participants. This can bedone using a heuristic algorithm or a machine learned algorithm. Withrespect to an example heuristic, the present invention can assignconfidence weightings to final predictions based on the following: aperson who makes a significant change of their Primary PredictionVariable to conform to a majority within their local sub-swarm is likelyless confident than someone who largely resists conforming. Also, aperson who increases their wager as a result of conforming to a majorityof the sub-swarm is less likely to have strong conviction than a personwho increases their wager by conforming to a minority within thesub-swarm. In addition, machine learning has been used to optimize thealgorithm that turns behavior into confidence levels. This is describedin the section labeled Machine Learning below.

In the next optional update input step 612, users optionally updateinput and user input data collected and stored during the secondaryforecasting round.

Next, in the final predication step 614, the central server now computesa final optimized aggregated prediction (also referred to as a finalcollaborative forecast). This aggregation is based on the full set ofinitial predictions, the full set of final predictions, the initialRound One weighting factors, and the Round Two weighting factorsgenerated from the behavioral data based on how people change. The finalcollaborative forecast provides an answer to the forecasting query (orqueries).

Optional additional round(s) step 616 may be included in the method. Thecentral server can repeat the process of steps 610 and 614 by displayingthe updated forecasts of sub-swarms to participants (that came out ofsteps 602 and 604) and enabling participants to update their forecastsagain. This enables further refinement of participant beliefs, andenable further capturing and assessment of behavioral data.

Referring next to FIG. 12, an exemplary volumetric visualization ofhyper-swarm characteristics 1200 is shown.

Referring again to FIGS. 1-12, because every participant is representedin a 3D grid structure wherein every participant evaluates the beliefsof their 26 neighbors and updates their own beliefs (in parallel), thefull behavior of the 1000-member population can be visualized and/oranalyzed as a complete system. In certain local regions, random noisemay distort results, while on most regions, consistent results generallyemerge. Volumetric visualization techniques can be used to display (a)the range of final predictions, (b) the range of confidence values, (c)the range of variability within sub-swarms, (d) the range of changebetween initial and final predictions within sub-swarms. An examplevolumetric visualization of hyper-swarm characteristics 1200 is shown inFIG. 12.

Typical polling systems for forecasting ask each individual to providean isolated forecast on a survey and computes a statistical aggregation.The problem is, every participant in that population has (a) a differentlevel of confidence and/or conviction in their answer, (b) they are verypoor at expressing or even knowing their true confidence on a survey,and (c) if asked to report their confidence, every individual has a verydifferent internal scale—so reported confidence can't be averaged acrossparticipants with accuracy. For prediction markets, rather than polls,there is generally a more authentic expression of confidence as peopleare putting money on the line, but because every transaction happens inseries, each person is influenced by the person who comes before them,who were in term influenced by the person who came before them. Thiscreates a “snowballing” effect that has been shown to significantdistort results. Thus, traditional methods fail, as they either (a)provide no interaction by which participants can converge on commonconfidence scales, or (b) enable interaction in series, which causessnowballing effects that amplify noise and reduce accuracy.

To solve this problem, the unique parallelized behavioral data iscollected in the steps of FIG. 6 above, enabling participants toevaluate the beliefs of others (forecast and confidence) withoutsnowballing effect. This, combined with unique machine learningtechniques, has enabled us to weight the contributions of eachparticipant in the population based on a more accurate indication oftheir relative confidence in their personal predictions, and/or therelative accuracy of those predictions.

For example, one basic embodiment of this process for estimatingconfidences using machine learning as follows: the system collectsbehavioral data that includes the (a) the magnitude of the change in theprimary forecast variable (e.g. who will win and by how much) betweenthat participants initial forecast and final forecast, (b) whether thenchange in primary forecast variable corresponds with closing the gap(i.e. compliance magnitude) between that participant and the majority orlargest plurality or statistical mean of the neighbor data points thatinfluenced that user or if it reflected resistance to conforming withthe neighbor data points (i.e. defiance magnitude). (c) the time takento adjust the sliders when considering the neighbor data points, (d) themagnitude of the change in the secondary forecast variable that reflectsconfidence (e.g. how much would you bet) between that participantsinitial forecast and final forecast (i.e. did they get more confident orless confident as a result of viewing the neighbor datapoints), (e)combined effects—did the user defy the neighbors but reduce confidence,or defy neighbors and increase confidence, or comply with neighbors andincrease confidence, or comply with neighbors and reduce confidence, andby how much, (f) and how did these behaviors vary across the full set offorecasts (e.g. the set of all football games predicted for a givenSunday)—as it is telling if a participant is defiant on some forecasts,and compliant on others—and so, the defiance vs compliance values arenormalized across the full set to assess their relative confidenceacross games.

Using this data, the machine learning system is trained on the trueoutcome of the primary forecast variable, with scoring scaled for eachuser by their secondary forecast variable (e.g. confidence). UsingFootball as an example, if the true outcome of the game is Team A+4, andthe user's initial guess is Team A+9, they might get an initial accuracyscore of: |(4−9)|=5, where the lower the score, the better, with aperfect score being 0. If their updated score was +8, they will get anupdated score of: |(4−8)|=4, where the lower the score, the better, witha perfect score being 0. The updated score is also normalized withconsideration as to whether the neighbors influenced towards the correctscore, or away from the correct score. These scores act as a measure ofthe user's skill in prediction overall, and by training a MachineLearning algorithm on these scores, the system can predict which usersare more likely to be most skillful at predicting the game in question.With this predicted skill level, the software of the system is then ableto weight new users' contributions to the hyper-swarm.

In some embodiments of the present invention, a plurality of values aregenerated for each participant that reflect that participant's overallcharacter across the set of events being predicted (e.g. a full set offootball games). By looking across the set of predictions, additionalcharacteristics can be generated. As described in co-pending patentapplications incorporated by reference, outlier index is one suchmulti-event value that characterizes each participant with respect tothe other participants within the population across a set of eventsbeing predicted. In addition, a confidence index is generated in someembodiments of the present invention as a normalized aggregation of theconfidence values provided in conjunction with each prediction withinthe set of predictions. For example, in the sample set of questionsprovided above, each prediction includes Confidence Question on a scaleof 0% to 100%. For each user, the confidence index is the averageconfidence the user reports across the full set of predictions, dividedby the average confidence across all users across all predictions in theset. This makes the confidence index a normalized confidence value thatcan be compared across users. In addition, multi-event self-assessmentvalues are also collected at the end of a session, after a participanthas provided a full set of predictions

In some embodiments of the present invention, a plurality of multi-eventcharacterization values are computed during the data collection andanalysis process including (1) Outlier Index, (2) Confidence Index, (3)Predicted Self Accuracy, (4) Predicted Group Accuracy, (5)Self-Assessment of Knowledge, (6) Group-Estimation of Knowledge, (7)Behavioral Accuracy Prediction, and (8) Behavioral ConfidencePrediction. In such embodiments, additional methods are added to thecuration step wherein Machine Learning is used to find a correlationbetween the multi-event characterization values and the performance ofparticipants when predicting events similar to the set of events.

In such embodiments, a training phase is employed using machine learningtechniques such as regression analysis and/or classification analysisemploying one or more learning algorithms. The training phrase isemployed by first engaging a large group of participants (for example500 to 1000 participants) who are employed to make predictions across alarge set of events (for example, 20 to 40 baseball games). For each ofthese 500 to 1000 participants, and across the set of 20 to 40 events tobe predicted, a set of values are computed including an Outlier Index(OI) and at least one or more of a Confidence Index (CI), a PredictedSelf Accuracy (PSA), a Predicted Group Accuracy (PGA), a Self-Assessmentof Knowledge (SAK), a Group Estimation of Knowledge (GAK), a BehavioralAccuracy Prediction (BAP), and a Behavioral Confidence Prediction (BCP).

In addition, user performance data is collected after the predictedevents have transpired (for example, after the 20 to 40 baseball gameshave been played). This data is then used to generate a score for eachof the large pool of participants, the score being an indication of howmany (or what percent) of the predicted events were forecast correctlyby each user. This value is preferably computed as a normalized valuewith respect to the mean score and standard deviation of scores earnedacross the large pool of participants. This normalized value is referredto as a Normalized Event Prediction Score (NEPS). It should be notedthat in some embodiments, instead of discrete event predictions, userpredictions can be collected as probability percentages provided by theuser to reflect the likelihood of each team winning the game, forexample in a Dodgers vs Padres game, the user could be required toassign percentages such as 78% likelihood the Dodgers win, 22%likelihood the Padres win. In such embodiments, alternate scoringmethods may be employed by the software system disclosed here. Forexample, computing a Brier Score for each user or other similar costfunction.

The next step is the Training Phase wherein the machine learning systemis trained (for example, using a regression analysis algorithm or aneural network system) to find a correlation between a plurality of thecollected characterization values for a given user (i.e. a plurality ofthe Outlier Index, the Confidence Index, a Predicted Self Accuracy, aPredicted Group Accuracy, a Self-Assessment of Knowledge, a GroupEstimation of Knowledge, a Behavioral Accuracy Prediction, and aBehavioral Confidence Prediction) and the Normalized Event PredictionScore for a given user. This correlation, once derived, is used by theinventive methods on characterization value data collected from newusers (new populations of users) to predict if the users are likely tobe a strong performer (i.e. have high normalized Event PredictionScores). In such embodiments, the machine learning system (for exampleusing multi-variant regression analysis) will provide a certainty metricas to whether or not a user with a particular combination ofcharacterization values (including an Outlier Index) is likely to be astrong or weak performer when making event predictions. In otherembodiments, the machine learning system will select a group ofparticipants from the input pool of participants that are predicted toperform well in unison.

Thus, the final step in the Optimization and Machine Learning process isto use the correlation that comes out of the Training Phase of themachine learning system. Specifically, the trained model is used byproviding as input a set of characterization values for each member of anew population of users, and generating as output a statistical profilefor each member of the new population of users that predicts thelikelihood that each user will be a strong performer based only on theircharacterization values (not their historical performance). In someembodiments the output is rather a grouping of agents that is predictedto perform optimally. This is a significant value because it enables anew population of participants to be curated into a high performingsub-population even if historical data does not exist for those newparticipants.

Referring next to FIG. 13, a flowchart for a real-time swarmingembodiment method is shown.

The real-time swarming method is structured as a real-time forecastingmethod wherein all participants engage in a synchronous process withintheir sub-swarms, with real-time data transfer within the limits ofhuman perceptual abilities. The steps are similar to the methods shownabove in FIG. 6, but is structured to enable continuous change ratherthan discrete rounds. This has profound advantages because it allows thefull population to function as a unified system, with data propagatingthrough the population as a result of every participant being a memberof an overlapping sub-swarm. In this way, all users are exposed to atotally unique set of other users (and their respective beliefs) greatlyreducing the possibility that random noise gets amplified (as happens inserial markets), while enabling the full system to still converge onoptimized results. The propagation of sentiments within a large-scalepopulation as it converges on a global sentiment can be visualized inreal time using the volumetric approach shown above with respect to FIG.12.

The process starts with an engagement process (Stage 1, step 1300)wherein participants are coordinated for a session that requiressynchronous activity. This is followed by an initial prediction stage(Stage 2, steps 1302 and 1304) wherein participants provide theirbaseline forecasts (i.e. the starting points for their participation).This initial prediction period allows for research into the questions,promoting information gathering. This is then followed by a real-timeinteractive updating (Stage 3, step 1308). A fixed time limit is placedupon this process, for example 60 to 90 seconds wherein participantsupdate their individual beliefs in real-time, based on exposure to thebeliefs of their neighbors. But unlike the prior embodiment, this iscontinually updated in real-time, thus participants see the dotsrepresentation the views of their neighbors change in real time (move inreal time) as everyone is updating their forecast synchronously. Butbecause every user only has a view into their individualized sub-swarm,unique beliefs can emerge in different places within the hyper-swarmstructure. And because all sub-swarms overlap, beliefs will propagateacross the structure, ideally until a global consensus (global maxima)is reached.

In the first computing device notification step 1300, as with the priorembodiment shown in FIG. 6, this present can be configured such that acentral server initiates a synchronous session by sending out a pushnotification to a plurality of computing devices, each of the computingdevices associated with one of a plurality of participants, each of thenotifications indicating that a new forecast is beginning. Thisnotification can be pushed to personal computers, tablets, or phones,and is generally done through a traditional software application, amobile app, or a web-app such that it pops up with a visual, audio,and/or tactile alert to get their attention.

In a preferred embodiment, each computing device receives an alertinforming them that a hyper-swarm session is starting and asking them tojoin the process. They can respond affirmatively or negatively byclicking a user interface (for example, a simple button response). Ifthey respond affirmatively, the central server will add them to a datastructure used for this prediction. In the preferred embodiment, thedata structure will assign each participant a coordinate within a 3Dgrid or an alternative data structure, as described above.

In the next initial forecasting round step 1302, similar to step 602,each participant in the hyper-swarm session is asked to make a set ofinitial individual forecasts, for example forecasts with respect to allof the football games to be played on a given day. In one exampleembodiment, two sliders appear on the screens of all participants foreach game forecast, as well as a global TIMER for the full set offorecasts. For a session involving forecasting all 15 football games fora given weekend, the participants may be given a 20 minute timer, forexample. This enables the participants to be coordinated in time, butalso gives enough time for participants to research their answers. Thesystem encourages research, as the more information participants gather,the smarter the overall hyper-swarm. The exemplary display of FIG. 7 isalso applicable to the current method.

In addition to the above sliders, the “INITIAL FORECAST COMPLETE” button706 is provided to participants, such that once they finish enteringtheir forecast, they can register the data as final. Or they can waitfor the timer to fully expire, at which point the data is alsoregistered as final. Either way, the final values for each of the userinterfaces 700, 702 above (Primary Forecast Variable user interface 700and Secondary Forecast Variable user interface 702) is sent to thecentral server and is stored associated with the given user and thatuser's coordinate in the grid structure. Other variables may also bestored, such as demographic data about the users age, gender, location,experience level on the subject, and self-reported skill level on thesubject in question. In some embodiments, each grid location is a blockin a blockchain structure, enabling secure data storage.

After the initial forecasting round 1302 is complete, the methodproceeds to the perform population curation analysis step 1304. Thecentral server now has initial forecasts from all participants in thissession. In the optional culling/weighting step 1306, using this data,the system performs a population curation analysis, determining whichparticipants are most likely to be skilled forecasters and whichparticipants are less likely to be skilled forecasters. This process isdescribed in co-pending U.S. application Ser. No. 16/059,698 by thepresent inventor, hereby incorporated by reference. The output of thisprocess is a weighting factor associated with each participant, theweighting factor indicative how likely that participants forecast valuesare accurate, with the most likely accurate forecasters being weightedhigher than the least likely. In some embodiments, a culling process isperformed in step 1306 and used in addition to or instead of theweighting process such that the weakest predicted performers are removedfrom the process. The culling and weighting processes of step 1306 aresimilar to those of step 606 previously described in FIG. 6.

The method then proceeds to the next secondary forecasting round updatedin real-time step 1308. The central server has now collected data fromall participants, optionally culled participants to a smaller set oflikely strong performers (in step 1306), and optionally weightedremaining participants based on the predicted likelihood that they arestrong performers (in step 1306). The current step 1308 is forparticipants to update their forecasts based on a stimulus. The stimulusis an indication to each participant of how other participants predictedthe same events. This happens in parallel (all participants are informedat substantially the same time) so there is not a sequential biasingproblem. In addition, all participants are provided with a unique butoverlapping stimulus set, to ensure that a diverse range of responses isgenerated by the population (i.e. to ensure the population is not allresponding to the same stimulus). This is where the sub-swarm definitioncomes in, as each participant in the defined data structure has a uniqueset of 26 neighbors. In this method, each participant is giveninformation to review about how their 26 neighbors forecast the sameevent compared to their own forecast. An example display is shown abovein FIG. 8.

As shown in FIG. 8 above, similarly to the method of FIG. 6, eachparticipant is provided with the inventive display of information anddata entry, enabling them to gain insight into the beliefs of a sub-setof fellow participants and update their own beliefs. Specifically, FIG.8 shows the two-dimensional coordinate grid 804 representing the twovariables provided by participants, with the Primary ForecastingVariable on the X axis 800 and the Secondary Forecasting Variable on theY axis 802. What is different about this method from the method of FIG.6, is that as each participant updates their sliders (in real time)their updated data is sent by the central server to all their neighborsin real time. Thus, for the participant who sees the image above in FIG.8, all of the neighbor forecast data points 810 representing other usersare moving in real time as those users are updating their beliefs basedon their unique set of neighbors. This puts the entire hyper-swarmstructure into motion in real-time, with feedback loops connecting allusers through overlapping sub-swarms. This will cause beliefs topropagate throughout the structure, battling for dominance until a localmaxima emerges. Or, until it is determined that no unified belief willemerge.

The displayed countdown timer 812 (starting at 60 seconds in thisexample) provides a time limit on how long this interactive process willcontinue. For a small swarm, 60 seconds may be enough, but for a verylarge-scale swarm, a longer period may be required.

In some interactive embodiments, a participant has the option ofchoosing an alternate data display. In one such embodiment, the userscan display a real-time histogram that represents the continuallychanging beliefs of their sub-swarm. This is most useful for embodimentsof much larger populations, for example 1 million users, represented ina 100×100×100 grid, and provided data about 7×7×7 neighbors (343neighbors). For such an example, the dots would be overwhelming, but ahistogram is informative as shown above in FIG. 11.

At the end of the interactive period, the central server has bothinitial forecasts and updated forecasts (captured as a sequence of timevarying values from all participants in this session) for each of theforecast events (i.e. each of the football games). Using this data, Inthe final step 1310, the central server can now perform a finaloptimization wherein the behaviors of users, performed in response tothe unique sub-swarm data displayed to them, is used to compute finalweighting factors indicative of an inferred confidence level ofparticipants. This can be done using a heuristic algorithm or a machinelearned algorithm as described previously.

Machine learning can follow the same basic model described above for thetwo-round system of FIG. 6 (using the initial and final forecasts foreach participant in the same manner). Machine learning can also be moresophisticated, as the full population is interacting at the same time,providing deeper behavioral data over time. In these embodiments,time-based characteristics are captured, assessed, and used for machinelearning, reflecting not just the magnitude of the changes in primaryand secondary forecast values, but the speed of change, delay untilchange, and timing of the change, across the convergence time period. Inaddition, the behavior of each participants forecast changes can beassessed in comparison the changes (in real time) of the neighborforecasts they are exposed to.

Referring next to FIG. 14, an exemplary grid-arranged population 1400 isshown. Thus far we have considered hyper-swarms comprised of randomlyselected populations that are uniform in distribution, with curationrelated to determined skill level in predictions. For hyper-swarms thatgenerate insights related to decisions or predictions, it is sometimesinformative to curate populations by demographic characteristics. Insome such embodiments, populations can be curated with informationregarding the age, gender, location, political affiliation, interests,and expertise of participants. In some such embodiments, the placementof individuals within the xyz grid structure can be controlled such thatrelatively even distributions of demographic characteristics fall withineach sub-swarm region of the grid. While it may be impossible generateperfectly even distributions, the large number of sub-swarms will allowthe minor fluctuations in a given demographic characteristic to cancelout. In other embodiments, it may be desirable to assign people ofparticular bias, expertise, or characteristic to certain regions of thexyz structure, enabling observation of how predictions, decisions, oropinions propagate through the structure.

For example, a political hyper-swarm could be structured with evendistributions of democrats and republicans across sub-swarms. Or, couldbe structured with segregated sub-swarms, with boundaries side by side,enabling very informative propagations of sentiment throughout thestructure. An example of such a structured arrangement is shown in FIG.14. The grid-arranged population 1400, arranged on the xyz coordinatesystem, includes a Republican region 1402 and a Democrat region 1404.

In some embodiments, arrangement of the hyper-swarm structure can beeven more complex, with different regions for example, for differentage-groups and/or genders and/or professions and/or politicalaffiliation and/or level of education. For sports predictions, regionscan be defined, for example, based on favorite sports, favorite teams,and/or experience level in predicting the sport in question. The valueof such a prescribed structure is that optimized solutions willpropagate throughout the hyper-swarm in informative ways, indicating ifa particular sentiment can emerge even when populations are segregatedby critical characteristics.

While it would be totally impractical to have 1000 people participate ina “chat room” to debate the issues, having real-time participants brokeninto sub-swarms can be supplemented with localized chat rooms fordiscussion and debate of the issues being forecast and/or decided. Whatis revolutionary about this architecture is that each sub-swarm is aunique distribution of people, all overlapping. This means a group of1000 people can have a conversation where ideas propagate throughout thefull population, but no single individual will interact with more thantheir 26 neighbors (a totally manageable number for real-timecommunication). Text chat can be structured as group chat, or can bestructured so that each participant has the ability to private messageany one of their neighbors, to ask for details as to why their opinionis the way it is on the chart.

This cannot easily be extended to voice chat among sub-swarms, withoverlapping distributions of people—because in voice, timing matters andyou can have multiple people talking at the same time, because they arenot in the same sub-swarm as each other, but are in your sub-swarm. Thiscan be inventively handled with (a) voice buffering to avoid overlap intime or (b) moderated turn-taking, but create complex logistics. Textchat solves this, as timing is not as consequential.

In some embodiments of the present invention, alternate data structurescan be used for representing the relationships between participants. Inthe embodiments above, the participants are structured in an xyz grid,with sets of neighbors called sub-swarms which overlap, enablingpropagation of influence across the full structure. To drive deeperpropagation, the structure can also include “hyper-neighbors” which areparticipants treated as being part of a given user's sub-swarm but whoare not local in the xyz grid. A “hyper-neighbor” could be someone whois a random distance away within the grid. This enables propagation ofsignals through jumps across the system. In many ways, this is hownetworks of neurons in neurological brains are structured, as mostinfluence is local neighbors, but some neurons bridge between regions.

One way to link regions is to have participants at critical node pointswithin the structure, participate in a unique sub-swarm (referred toherein as a supervisory-subswarm) that only includes other node points.These participants are therefore acting as a higher order aggregatorwithin the system, as they are seen as neighbors to their local regionwithin the structure, but they see as neighbors participants who are atother node points. FIG. 15 shows an exemplary supervisory subswarmgrid-arranged embodiment 1500 including node participants 1502, spacedevenly across the larger xyz structure (hidden node participants 1502 atthe center and back faces are not shown for clarity). Conventionalparticipants 1504 located at other grid intersections as previouslydescribed.

In the embodiment above, the members of the supervisory subswarm can beprovided with two data displays at once for use in making their updatedpredictions. One data display shows them their local neighbors withinthe xyz grid, and one data display that shows them the hyper-neighborsat the distant node points, thus giving them a view into the views ofthe system as a whole. For a real-time system that is converging inparallel, these supervisory members enable rapid propagation ofinformation. In some embodiments, supervisory members are identified toall members within their local display, so they can see the views of thesupervisory member in their evaluation, for example by having their datadot a unique color or size.

In some embodiments, members of supervisory subswarms are weightedhigher in the final aggregation process, as they have considered a moreglobal view of the full population, as the full population hasconverged.

Referring next to FIG. 16, an exemplary slider set user interface for amulti-option question is shown in one embodiment of the presentinvention.

The above examples have been presented using a user interface presentingat least one forecast variable that is a linear scale between twooutcomes (e.g. outcomes of Team A wins and Team B wins). The presentinvention can support a wide range of other question types. For example,a multi-option question such as: “Who will win: A, B, C, D, or E?”. Anexample of this type of question could be “Who will win Best Actor inthe Oscars: Actor A, Actor B, Actor C, Actor D, or Actor E?” To supportthis type of question, a set of slider-based user interfaces can beprovided, asking participants to give a probability for each answeroption, each slider user interface linked so that they must add to 100%.An exemplary display for this multi-option question is shown in FIG. 16.Five user interfaces are shown: an option A interface 1602, an option Buser interface 1604, an option C user interface 1606, an option D userinterface 1608, and an option E user interface 1610.

In the example shown in FIG. 16, the Primary Forecast Variable for eachparticipant has 5 values (P1, P2, P3, P4 and P5). The participant movesthe slider associated with each user interface 1602, 1604, 1606, 1608,1610 to select a percentage value for each option. The slider values arelinked such that P1+P2+P3+P4+P5=100%. In addition to the user interfacesfor the primary forecast variable, a set of Secondary Sliders can beprovided (not shown) that enable the participant to indicate confidencein each of these forecasts, either individually, or holistically (forthe whole set), or both.

To implement the multi-option method, we need to enable participants tosee a visual representation of how their forecast across the fiveprimary values compares with their neighbors in their sub-swarm. Anexample inventive visualization display 1700 is shown in FIG. 17, whereeach participant input is represented by subswarm participant dots 1702.Each column 1704 is associated with one of the primary forecastvariables, in this example the same question and variables shown in FIG.16. The subswarm participant dots 1702 are arranged in each column 1704to present how the sub-swarm participants distributed their fiveprediction probabilities. The prediction probabilities of theparticipant are indicated by the larger dots 1706 showing thatparticipant how their forecast compares with the other forecasts fromtheir sub-swarm across options A, B, C, D, and E.

In the example above, the preferred embodiment enables the participantto “grab” each large dot 1706 that represent their forecast for each ofthe five items in this example, and slide the dot up or down to adjusttheir forecasts. This is clear and intuitive, and is easily implementedfor either (a) the iterative embodiment described with reference to FIG.6, and (b) the real-time embodiment described with reference to FIG. 13.In addition, behavioral data can be captured and used in machinelearning as described above, for this multi option method. In fact, thebehavioral data is even richer.

In the real-time embodiment, participants view the small subswarmparticipant dots 1702 moving in real time, as they adjust their personalparticipant dots 1706.

This allows convergence on optimal solutions with feedback loops,locally, while overlapping swarms allow propagation throughout the fulldata structure.

In the above example, a single confidence slider user interface canadditionally be implemented in some embodiments. In a preferredembodiment, the single confidence slider user interface (i.e. thesecondary variable) is provided to indicate and theoretical wager ontheir top choice. For example—“How much would you bet on your top choicewinning the Oscar?” This data provides a scaling factor the relativeconfidence across all choices and can be used in the processes describedabove.

Finally, the methods described herein can be used to provide insightfulvolumetric visualization of each of the Primary variables (P1, P2, P3 .. . ) being forecast in this way. For example, each of the forecastoptions can be viewed in real time, as the sentiment propagates acrossthe grid structure.

Another innovative method developed herein is called “neighborsuffering” or “subswarm shuffling” and it involves changing the relativelocations of participants in the grid structure over time. There are twounique methods that have been developed.

Firstly, in a “Shuffling Between Predictions” method, the grid israndomized (either completely or partially) between predictions in aprediction set, to ensure that any random biases within sub-swarms arecanceled out across a set of predictions. For example, if the predictionset includes 15 football games, the subswarms can be completely orpartially randomized between the predictions of each game. This meansthat when participants view their neighbors' predictions in a sub-swarm,those neighbors will be a completely or partially different set ofparticipants for each of the 15 predictions.

Secondly, in a “Shuffling Between Rounds in a Multi-Round iterativeprediction” method, as described above, a two round prediction isdescribed. That said, in some embodiments, additional rounds can beimplemented, as described above. To make those additional rounds morevaluable, the subswarm compositions can be updated in each additionalround. In other words, the grid is randomized (either completely orpartially) between additional rounds, to ensure that any random biaseswithin sub-swarms are canceled out across rounds. For example, if theprediction set includes four rounds, the subswarms can be completely orpartially randomized between Round 2 and Round 3, and between Round 3and Round 4. This means that when participants view their neighborspredictions in a sub-swarm for those rounds, those neighbors will be adifferent set of participants for each of the two subsequent rounds.

While many embodiments are described herein, it is appreciated that thisinvention can have a range of variations that practice the same basicmethods and achieve the novel collaborative capabilities that have beendisclosed above. Many of the functional units described in thisspecification have been labeled as modules, in order to moreparticularly emphasize their implementation independence. For example, amodule may be implemented as a hardware circuit comprising custom VLSIcircuits or gate arrays, off-the-shelf semiconductors such as logicchips, transistors, or other discrete components. A module may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices or thelike.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions that may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code could be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork.

While the invention herein disclosed has been described by means ofspecific embodiments, examples and applications thereof, numerousmodifications and variations could be made thereto by those skilled inthe art without departing from the scope of the invention set forth inthe claims.

What is claimed is:
 1. A method for real-time computer-moderatedcollaborative forecasting among a population of human participants usinga plurality of networked computing devices, the method comprising:providing a collaboration server running a collaboration application,the collaboration server in communication with the plurality of thenetworked computing devices, each computing device associated with oneparticipant; providing a local forecasting application on each networkedcomputing device, the local forecasting application configured fordisplaying forecasting information to and collecting forecasting inputfrom the one participant associated with that networked computingdevice; and enabling through communication between the collaborationapplication running on the collaboration server and the localforecasting applications running on each of the plurality of networkedcomputing devices, the following steps: send a forecasting query to theplurality of networked computing devices, the forecasting querydescribing a future event to be collaboratively predicted by thepopulation of human participants; present, at substantially the sametime, a representation of the forecasting query to each participant on adisplay of the computing device associated with that participant;collect an initial forecast response from a plurality of theparticipants via a user interface on the computing device associatedwith that participant; store each collected initial forecast response ina unique location in a data structure in a memory accessible by thecollaboration server, wherein each initial forecast response isassociated with the participant the response was collected from;identify, for each of at least two of the plurality of the participants,a subset of the initial forecast responses stored in unique locations inthe data structure of the set of initial forecast responses, whereineach subset of the initial forecast responses corresponds to a uniquesubset of locations in the data structure and has a unique membership ofthe set of collected initial forecast responses and wherein each subsetincludes at least one initial forecast response that is also included inat least one other subset of initial forecast responses; display, toeach participant for which a subset is identified, a graphicalrepresentation of the identified subset of initial forecast responsesassociated with that participant, wherein the graphical representationis displayed on the computing device associated with that participant,whereby each participant is presented with a unique graphicalrepresentation corresponding to the unique subset of locations of theidentified subset of initial forecast responses associated with thatparticipant; after displaying the graphical representation of the subsetof initial forecast responses to each participant, collect an updatedforecast response from each participant for which a graphicalrepresentation is displayed via the user interface on the computingdevice associated with that participant; store a set of updated forecastresponses from the plurality of participants for which the graphicalrepresentation is displayed in a memory accessible by the collaborationserver; and compute a final collaborative forecast based at least inpart upon the set of initial forecast responses and the set of updatedforecast responses, the collaborative forecast providing an answer tothe forecasting query.
 2. The method of claim 1 wherein computing thefinal collaborative forecasting includes assessing the change, for eachparticipant for which an updated forecast response is received, betweenthe initial forecast response they provided and the final forecastresponse they provided.
 3. The method of claim 1 further including thestep of assigning a unique sub-population to every one of the pluralityof participants, wherein each sub-population is a unique subset of thefull population of networked human participants that shares at least oneparticipant with at least one other sub-population.
 4. The method ofclaim 3 wherein the subset of initial forecast responses for each of theplurality of participants consists of the initial forecast responses ofthe sub-population assigned to that participant.
 5. The method of claim1 wherein the steps of presenting the forecast query, collecting updatedforecast responses, and storing the set of collected responses arerepeated multiple times prior to the step of computing the finalcollaborative forecast, wherein a plurality of sets of updated forecastresponses are stored over a time period, and wherein the finalcollaborative forecast is based at least in part upon the plurality ofsets of updated forecast responses stored over the time period.
 6. Themethod of claim 1 further comprising, during the step of collecting theupdated forecast responses, of displaying of a countdown timer on thedisplay associated with each participant indicating an amount of timeleft for collaborative forecasting, the countdown timer for eachparticipant substantially synchronized.
 7. The method of claim 1 whereinthe graphical representation of the subset of initial forecast responsesincludes a graphical histogram including the subset of initial forecastresponses.
 8. The method of claim 1 wherein the graphical representationof the subset of initial forecast responses includes a set of graphicaldots wherein each graphical dot represents one of the subset of initialforecast responses.
 9. The method of claim 1 further comprising, duringdisplaying of the graphical representation of the identified subset,also displaying to each participant a graphical indicator showing theparticipant's own current forecast response in relation to the displayedsubset of identified subset of initial forecast responses.
 10. A methodfor real-time computer-moderated collaborative forecasting among apopulation of human participants using a plurality of networkedcomputing devices, the method comprising: providing a collaborationserver running a collaboration application, the collaboration server incommunication with the plurality of the networked computing devices,each computing device associated with one participant; providing a localforecasting application on each networked computing device, the localforecasting application configured for displaying forecastinginformation to and collecting forecasting input from the one participantassociated with that networked computing device; and enabling throughcommunication between the collaboration application running on thecollaboration server and the local forecasting applications running oneach of the plurality of networked computing devices, the followingsequential steps: send a forecasting query to the plurality of networkedcomputing devices, the forecasting query describing a future event to becollaboratively predicted by the population of human participants;present, at substantially the same time, a representation of theforecasting query to each participant on a display of the computingdevice associated with that participant; collect, in real-time for eachof a plurality of time steps comprising an initial forecast period,initial forecast responses from each participant via a user interface onthe computing device associated with that participant; store, inreal-time for each time step, each initial forecast response in a uniquelocation in a data structure in a memory accessible by the collaborationserver, wherein each initial forecast response is associated with theparticipant the response was collected from; identify, in real-time foreach time step, for each of at least two of the plurality ofparticipants, a subset of the initial forecast responses stored inunique locations in the data structure of the set of initial forecastresponses associated with that time step, wherein each subset of theinitial forecast responses corresponds to a unique subset of locationsin the data structure and has a unique membership of the set ofcollected initial forecast responses and wherein each subset includes atleast one initial forecast response that is also included in at leastone other subset of initial forecast responses; display, for each timestep, at substantially the same time for each participant, a graphicalrepresentation of the identified time-step subset of initial forecastresponses associated with that participant on the computing deviceassociated with that participant, whereby each participant is presentedwith a unique graphical representation corresponding to the uniquesubset of locations of the identified subset of initial forecastresponses associated with that participant; after displaying thegraphical representation of the subset of initial forecast responses toeach participant, in real-time for each of a plurality of time stepscomprising an updated forecast period, collecting an updated forecastresponse from each participant via the user interface on the computingdevice associated with that participant; store, in real-time for eachtime step, a set of updated forecast responses from the population ofhuman participants in a memory accessible by the collaboration server;and compute a final collaborative forecast based at least in part uponthe sets of initial forecast responses and the sets of updated forecastresponses, the collaborative forecast providing an answer to theforecasting query.